A Query Processing Framework based on Hadoop

With the development of cloud computing and big data, the massive volume of dataset proposes a big challenge for cloud data management systems. Unlike traditional database management method, cloud data queries are typically parallel and distributed. Intuitively, the query processing framework should embrace these characteristics. In this paper, by leveraging the inherent data structure of Hadoop HDFS, we design a query process framework. Specifically, facilitated by the key-value structure of HDFS, we construct a two-level index which first locates the target nodes for desired data, and then search within each node for further combination. As for joint operation, our query process engine optimizes by judging if the join key is equal to index key. If not, a MapReduce based join algorithm is then called. In this way, our method can reduce the cost of query processing. Besides, we conduct experiments for empirical evaluation.